Systems and methods for debiasing media creative efficiency

ABSTRACT

A quantification system is configured for debiasing media creative efficiency. In some embodiments, the quantification system leverages a weighted generalized linear model (GLM) to determine the individual impacts of media creatives beyond network effects. To prepare input data for fitting the weighted GLM, the quantification system analyzes spot airing data, creates a specific data structure for storing observations (e.g., network-media creative combinations) that can be provided to the weight GLM as input, and computes additional input data points needed by the weighted GLM (e.g., network spend, media creative efficiency per network-media creative combination, etc.). The weighted GLM is then fitted to obtain coefficients representing the individual impacts of the media creatives. The quantification system utilizes the computed impacts to adjust the previously computed media creative efficiency for each media creative. In this way, relative performance of media creatives can be objectively quantified across networks without needing digital evidence.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims a benefit of priority under 35 U.S.C. § 119(e)from the filing date of U.S. Provisional Application No. 62/647,402,filed on Mar. 23, 2018, entitled “SYSTEMS AND METHODS FOR DEBIASINGMEDIA CREATIVE EFFICIENCY,” the entire disclosure of which is fullyincorporated by reference herein for all purposes.

TECHNICAL FIELD

This disclosure relates generally to data processing for performanceanalysis. More particularly, this disclosure relates to systems,methods, and computer program products for debiasing media creativeefficiency.

BACKGROUND OF THE RELATED ART

With the advent of the Internet, many aspects of modern life are nowdigitally connected through the seemingly ubiquitous smart phones, smarttelevisions (TV), smart home appliances, Internet of Things (IoT)devices, websites, mobile apps, etc. Even so, many more analog aspectsremain disconnected from this digital world. Linear TV is an example ofan offline medium that is disconnected from the digital world.

“Linear TV” refers to real time (live) television services that transmitTV program schedules. Almost all broadcast TV services can be consideredas linear TV. Non-linear TV covers streamlining and on-demandprogramming, which can be viewed at any time and is not constrained byreal-time broadcast schedules. Video-on-demand (VOD) and nearvideo-on-demand (NVOD) transmissions of pay-per-view programs overchannel feeds are examples of non-linear TV.

Because linear TV is an offline medium, it is not possible toautomatically collect information on viewers of linear TV. This createsa data gap problem. To address this data gap program, Nielsen MediaResearch, an American firm headquartered in New York, N.Y., U.S.A.,devised a ratings system, known as the Nielsen ratings, to determine theaudience size and composition of television programming in the UnitedStates. This determination is based on audience response to TV programsgathered in one of two ways—using viewer diaries or set meters attachedto TVs in selected homes. The former requires a target audienceself-record their viewing habits. The latter requires a special deviceto collect specific viewing habits on a minute to minute basis and sendthe collected information to Nielsen's ratings system over a phone line.Today, Nielsen's ratings system is the primary source of audiencemeasurement information in the television industry. Television networksrely heavily on the Nielsen ratings to decide the value of televisionshows.

While Nielsen's ratings system can provide some quantified measures ofaudience response to TV programs, the Nielsen ratings do not measureconversion rates for TV commercials. Accordingly, a typical approach forevaluating the performance of a TV commercial is to define theefficiency (E) of that TV commercial as Response per Amount Spent whereE is defined as E=100*lift/(ad spend). In this case, “lift” is a qualitymetric for measuring a TV commercial in the context of a particular typeof campaign—in this case, for measuring how much increase (lift) per$100 ad spend. This approach is generally independent to where and whenthe TV commercial aired on television networks.

SUMMARY OF THE DISCLOSURE

The approach described above can be troublesome in some cases. Forinstance, a media creative (which refers to content created for aparticular purpose or campaign) can be aired in multiple ad spots, ondifferent television networks (hereinafter “networks”), at differenttimes, under different cost structures. Because linear TV is an offlinemedium, it is not possible to build an actual rigorous test/designframework that eliminates biases based on digital evidence collectedfrom TV viewers and TV networks. Thus, if the level of efficiency of amedia creative (referred to herein as “media creative efficiency”) iscalculated using the same equation as a single ad spot (i.e., using theaggregated lift and aggregated ad spends), the performance of the mediacreative will be influenced by the performance of network, time, coststructure, and other factors. These influences represent biases in thedata which could lead to a false conclusion.

An object of this disclosure is to provide a solution for debiasing suchbiases and producing a true media creative efficiency that moreobjectively quantifies the relative performance of a media creative overother media creatives across multiple networks. According toembodiments, this object can be realized in a quantification systemhaving a media creative performance analyzer configured for debiasingmedia creative efficiency.

In some embodiments, the media creative performance analyzer leverages aweighted generalized linear model (GLM) to determine the individualimpacts of media creatives beyond network effects. To prepare input datafor fitting the weighted GLM, the media creative performance analyzer isoperable to analyze spot airing data, create a data structure forstoring observations (e.g., network-media creative combinations)suitable as input to the weight GLM, and compute additional inputsneeded by the weighted GLM (e.g., network spend, media creativeefficiency per network-media creative combination, etc.). The weightedGLM is then fitted to obtain a set of coefficients representing theindividual impacts of the media creatives. The media creativeperformance analyzer is further operable to adjust the previouslycomputed media creative efficiency for each media creative utilizing thecomputed individual impacts. In this way, relative performance of mediacreatives can be objectively quantified across networks without needingdigital evidence.

In some embodiments, a method of debiasing media creative efficiency caninclude retrieving spot airing data from a database, the spot airingdata comprising information on media creatives and networks on which themedia creatives aired; determining, based on the spot airing data, a setof media creatives with overlapping networks on which the set of mediacreatives aired; computing a media creative efficiency for each mediacreative of the set of media creatives per each network of theoverlapping networks; creating a data structure, each entry in the datastructure representing a network-media creative combination derived fromthe set of media creatives and the overlapping networks; modifying thedata structure to include a network spend for the each entry; computinga media creative efficiency for the each entry in the data structure;performing a weighted GLM fitting operation over the data structure toobtain impacts of the set of media creatives beyond effects of theoverlapping networks; and adjusting the media creative efficiency forthe each entry in the data structure utilizing the impacts of the set ofmedia creatives to thereby debias the media creative efficiency for theeach entry in the data structure. In some embodiments, categoricalvalues associated with the overlapping networks and/or any factor (e.g.,“rotation,” “times of day”, “geolocation”, etc.) under consideration bythe media creative performance analyzer are transformed into numericalvalues prior to performing the weighted GLM fitting operation.

In some embodiments, the method may further comprise determiningeligible media creatives for debiasing by creating a definition of nodesrepresenting the media creatives in the spot airing data and edgesrepresenting the networks on which the media creatives aired; creating adefinition of islands, each island representing a subgraph of the nodesconnected directly or indirectly through the edges; determining whetherany island is present in the media creatives; and responsive to anisland being present in the media creatives, returning a set of nodesassociated with the island as the set of media creatives.

In some embodiments, the method may further comprise automaticallydecoupling datasets in the spot airing data into non-overlappingclusters of groups of networks and media creatives and, for eachcluster, performing the weighted GLM fitting operation to obtainindividual impacts of media creatives and adjusting a media creativeefficiency previously computed for each media creative in the cluster tothereby debias the media creative efficiency for the each media creativein the cluster.

In some embodiments, the method may further comprise generating avisualization for presentation on a user device, the visualizationshowing the relative performance of different media creatives across thenon-overlapping clusters on a per cluster basis. In some embodiments,the visualization may be generated to show the relative performance ofthe media creatives in only one of the non-overlapping clusters (e.g.,the largest cluster). In some embodiments, the visualization may begenerated to show the relative performance of all the media creatives inthe same campaign across multiple networks.

One embodiment may comprise a system having a processor and a memory andconfigured to implement the method disclosed herein. One embodiment maycomprise a computer program product that comprises a non-transitorycomputer-readable storage medium which stores computer instructions thatare executable by a processor to perform the method disclosed herein.Numerous other embodiments are also possible.

These, and other, aspects of the disclosure will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. It should be understood,however, that the following description, while indicating variousembodiments of the disclosure and numerous specific details thereof, isgiven by way of illustration and not of limitation. Many substitutions,modifications, additions and/or rearrangements may be made within thescope of the disclosure without departing from the spirit thereof, andthe disclosure includes all such substitutions, modifications, additionsand/or rearrangements.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the disclosure. It should be notedthat the features illustrated in the drawings are not necessarily drawnto scale. A more complete understanding of the disclosure and theadvantages thereof may be acquired by referring to the followingdescription, taken in conjunction with the accompanying drawings inwhich like reference numbers indicate like features.

FIG. 1 depicts a diagrammatic representation of a quantification systemresiding on a network server communicatively connected to a variety ofdata sources according to some embodiments.

FIG. 2 depicts a diagrammatic representation of a quantification systemoperating in a network environment according to some embodiments.

FIG. 3 is a flow chart illustrating a method for debiasing mediacreative efficiency according to some embodiments.

FIG. 4A is a flow chart illustrating a method for preparing input datafor analysis according to some embodiments.

FIG. 4B is a flow chart illustrating a method for handling special casesthat may occur in the input data according to some embodiments.

FIG. 5 is a flow chart illustrating a method for converting impacts ofmedia creatives to adjusted efficiency for each media creative accordingto some embodiments.

FIG. 6 depicts a diagrammatic representation of an example of avisualization of debiased media creative efficiencies according to someembodiments.

FIG. 7 depicts a diagrammatic representation of a data processing systemfor implementing a system according to some embodiments.

DETAILED DESCRIPTION

The disclosure and various features and advantageous details thereof areexplained more fully with reference to the exemplary, and thereforenon-limiting, embodiments illustrated in the accompanying drawings anddetailed in the following description. It should be understood, however,that the detailed description and the specific examples, whileindicating the preferred embodiments, are given by way of illustrationonly and not by way of limitation. Descriptions of known programmingtechniques, computer software, hardware, operating platforms andprotocols may be omitted so as not to unnecessarily obscure thedisclosure in detail. Various substitutions, modifications, additionsand/or rearrangements within the spirit and/or scope of the underlyinginventive concept will become apparent to those skilled in the art fromthis disclosure.

As alluded to above, while Nielsen's ratings system can provide somequantified measures of audience response to TV programs, the Nielsenratings do not measure conversion rates for TV commercials. This is, inpart, because there is a natural distinction between online data andoffline data.

As illustrated in FIG. 1, online data and offline data can come fromvery different data sources. Further, data aggregated and/or provided bythese different data sources can have very different data types. When aTV commercial (which is an example of a media creative) is presented toan audience through an online medium (e.g., a website or a mobileapplication acting as an advertising channel), the online medium can bean effective tool for a server that is programmed to compute theperformance efficiency of that TV commercial. This is because theaudience (i.e., potential consumers of a product or service offered bythe TV commercial) is already accessing the Internet through the websiteor the mobile application. When a user is attracted to or interested inthe product or service and visits the website or uses the mobileapplication to access the product or service, there is a sessionassociated with that advertising channel. Thus, whether such a sessionresults in a sale (or conversion) is a relatively straightforwardprocess that can be done through tracking the session at the website orthe mobile application (e.g., using a tracking pixel embedded in a pageor pages that sends data to a server computer hosting the website or onethat is associated with an entity offering the product or service).

The offline medium (e.g., linear TV), on the other hand, aims to drivepotential consumers first to the Internet and then to a website orapplication where a product or service is offered. Unlike the onlinemedium, there is neither session tracking nor a direct relationshipbetween the offline medium and the desired result. Without digitalevidence collected from TV viewers and TV networks, it is not possibleto directly assess the performance efficiency of a media creative. Thisdisconnect makes it extremely difficult to truly measure the performanceof a media creative.

As discussed above, an alternative approach is to define the efficiency(E) of a media creative under evaluation as a measure of lift relativeto ad spend where E=100*lift/(ad spend). However, this approach assumesthat the performance efficiencies of media creatives, which aired ondifferent television networks, at different times, under different coststructures, are not affected by how, when, or how much they may differfrom one another. This assumption could lead to a false conclusion.Table 1 below illustrates this scenario.

TABLE 1 Average Lift Per Airing Network CA CB NA 100 120 NB 50 60Efficiency 9.5 6.6

Table 1 shows two media creatives (CA, CB) aired on two TV networks (NA,NB). As an example, a “media creative” can refer to a file (e.g., avideo, an audio, a multimedia file, etc.) that was created to convey amessage and/or for a particular purpose. An “ad spot” or “spot” canrefer to a media creative that airs at a particular time on a particularnetwork. In this example, each media creative aired 10 times (10 spots).However, CA aired 9 times on NA and 1 time on NB, while CB aired 1 timeon NA, and 9 times on NB. Suppose both NA and NB charges $1000 perairing. Using the standard approach where E=100*lift/(ad spend), theaggregated results are as follows:Efficiency(CA)=100*total_lift/total_spend=100*(100*9+50*1)/(10*1000)=9.5Efficiency(CB)=100*total_lift/total_spend=100*(120*1+60*9)/(10*1000)=6.6

The calculated results show that CB performed worse than CA (i.e.,Efficiency (CB)<Efficiency (CA)). This is because CB aired 90% times onNB and NB provides an average lift per airing that is lower than NA.However, as shown in Table 1, CB actually performs better than CA acrossany TV network. That is, the TV networks themselves are a factor thatcan skew the impact of the media creatives that they air, making itextremely difficult to calculate the true performance (e.g., efficiencyand effectiveness) of these media creatives.

For an accurate comparison among media creatives, a new solution isneeded to account for these differences (e.g., different TV networks,distribution of viewers, etc.) and eliminate biases that may falselyinfluence TV creative performance analyses. To this end, embodimentsdisclosed herein provide a solution that can quantitively debiaspossible biases in spot airing data and compute true media creativeefficiency.

As exemplified in FIG. 2, in some embodiments, this solution can berealized in quantification system 250 operating in network environment200. Quantification system 250 can be communicatively connected to spotairing data providers 210 a . . . 210 n through analog communicationchannels (e.g., telephones, mail, couriers, etc.). Quantification system250 can also be communicatively connected to spot airing data providers210 a . . . 210 n over network 220 (e.g., Internet). Examples of spotairing data providers 210 a . . . 210 n can include TV networks, mediaagencies, third-party data providers such as a market research firm,etc. Quantification system 250 may be implemented on one or more servermachines operated by a media performance analytics service provider. Themedia performance analytics service provider may purchase spots from TVnetworks to air media creatives. This relationship may providequantification system 250 with access to ad spend information which, inturn, enables quantification system 250 to compute efficiency for eachmedia creative per each network. Additionally or alternatively, spotairing data providers 210 a . . . 210 n can provide online data and/oroffline data, as exemplified in FIG. 1. Examples of offline data caninclude spot airing logs (before and after spots have aired) and ratesfrom TV networks, spot airing data, program schedules, programdemographics, etc.

Spot airing data generally includes what and when spots aired and onwhat network. Often there is not a uniform format of spot airing datareceived or obtained from spot airing data providers 210 a . . . 210 n.Quantification system 250 is operable to uniquely identify and storespot airing data per each instance of a spot airing on a particularnetwork at a particular time in spot airing data database 260. In oneembodiment, quantification system 250 is operable to perform, wherenecessary, data cleansing operations such as deduplication,normalization, data format conversion, etc.

In the example of FIG. 2, quantification system 250 further includesmedia creative performance analyzer 280 and visualizer 270. As describedbelow, media creative performance analyzer 280 is configured fordebiasing media creative efficiency so that a more accuracy mediacreative efficiency can be presented on, e.g., on client devices 230 a .. . 230 n through a user interface (UI) generated by visualizer 270.

FIG. 3 is a flow chart illustrating a method for debiasing mediacreative efficiency. To remove biases from potential factors that mayinfluence or otherwise affect the efficiency among media creatives, themedia creative performance analyzer (e.g., media creative performanceanalyzer 280) implements a new framework which leverages the concept ofa weighted generalized linear model (GLM). In statistics, the GLM is aflexible generalization of ordinary linear regression that allows forresponse variables that have error distribution models other than anormal distribution. The weighted GLM requires an input format that isvery different from spot airing data obtained or received by thequantification system (e.g., quantification system 250). Accordingly, insome embodiments, the media creative performance analyzer first preparesinput data to the weighted GLM (301). This preparation is furtherdescribed below with reference to FIGS. 4A and 4B.

As illustrated in FIG. 4A, preparing input data for analysis can includeaccessing a spot airing data database (e.g., spot airing data database260) and retrieving media creative data sets from the spot airing datadatabase (401). As discussed above, each data set can correspond to aspot of a media creative and can include an ad spot identifier (ID), amedia creative ID, a time when the spot was aired, on what network thespot was aired, how much the network charged for its airing (“spend”),etc.

In some embodiments, the media creative performance analyzer can includea function for automatically checking and handling decoupled scenarios.In this case, “decoupled” refers to the delineation that seems to dividethe media creative data sets. Table 2 below illustrates an example of adecoupled scenario. In the example of Table 2, five media creativesCA-CE aired on five networks NA-NE. These media creatives were createdfor the same purpose (e.g., for the same TV campaign) and are meant toconvey the same message.

TABLE 2 Media Creative Network CA CB CC CD CE NA $10K $3K 0 0 0 NB 0 $5K $4K 0 0 NC $12K 0 $10K 0 0 ND 0 0 0 $11K $8K NE 0 0 0  $7K $7K

From Table 2, it can be seen that not all media creatives were aired oneach network. In fact, the media creatives and the networks can begenerally partitioned along ad spend, as shown in Table 3 below.

TABLE 3 Media Creative Network CA CB CC CD CE NA $10K $3K 0 0 0 NB 0 $5K $4K 0 0 NC $12K 0 $10K 0 0 ND 0 0 0 $11K $8K NE 0 0 0  $7K $7K

As a result, a cluster of a group of media creatives (CA-CC) and a groupof networks (NA-NC) seems to be separable from another cluster of adifferent group of media creatives (CD-CE) and a different group ofnetworks (ND-NE). These two clusters have no overlapping networks and/ormedia creatives in common and thus are considered “decoupled” from oneanother.

This lack of overlapping network spend between the two decoupledclusters can affect the ability of the media creative performanceanalyzer to compare relative efficiencies between them. One reason isbecause the media creative performance analyzer focuses on identifyingrelative performance on overlapping networks. Accordingly, in someembodiments, the media creative performance analyzer is operable toautomatically identify possible decoupled scenarios in the mediacreative data sets. This can entail determining media creatives thataired on overlapping networks (and thus have network spend values thatcan be compared) (403). If no overlapping networks could be found in themedia creative data sets retrieved from the spot airing data database(405), the media creative performance analyzer may terminate the process(and/or proceed to process the next set of media creative data setscreated for another TV campaign) or proceed to FIG. 4B.

If media creatives with overlapping networks can be found, the mediacreative performance analyzer is operable to determine the efficiency(E) for each media creative per Network (407). In some embodiments, themedia creative performance analyzer is operable to create a datastructure for storing network-creative combinations (409). In someembodiments, this data structure, which can be created at anyappropriate time, is particularly structured for input to a weightedGLM. Before describing this data structure, an example of networks andmedia creatives under consideration by the media creative performanceanalyzer at this time may be helpful.

In the example of Table 4, media creative CA aired on network NA onceand creative CB aired once on network NA and once on network NB. Here,“network” is the only potential factor. However, it can be generalizedto include any potential factors such as “rotation”, “times of day”,“geolocation”, etc.

TABLE 4 Ad spot ID Creative ID Network Spend Lift 1 CA NA 100 50 2 CB NA100 40 3 CB NB 200 100

Using the sample networks and media creatives from Table 4, an exampleof a new data structure can be created as shown in Table 5 below:

TABLE 5 Media Creative Network CA NA CA NB CB NA CB NB

In some embodiments, this new data structure is specifically created forinput to the weighted GLM. In some embodiments, each entry in the datastructure represents an observation that is a combination of a network(and/or any factor under consideration) and a media creative (which, inthis example, is referred to as a network-media creative combination).

The data structure is then modified to include a column for representingthe ad spend per each network-creative combination (411). This isexemplified in Table 6 below.

TABLE 6 Media Creative Network Spend CA NA . . . CA NB . . . CB NA . . .CB NB . . .

Based on the ad spend per network-media creative combination, the mediacreative performance analyzer is operable to compute an efficiency foreach network-media creative-ad spend combination (413). In someembodiments, with sufficient samples, a network-media creativecombination with a “spend” value less than $250 is not considered.

The media creative performance analyzer is further operable to updatethe data structure to include the computed efficiency for eachnetwork-media creative-ad spend combination. This is exemplified inTable 7 below.

TABLE 7 Media Creative Network Efficiency Spend CA NA . . . . . . CA NB. . . . . . CB NA . . . . . . CB NB . . . . . .

In this example, “network” is a factor being considered by the mediacreative performance analyzer for debiasing. To consider such a factor,which is categorical rather than numerical, the media creativeperformance analyzer is operable to transform the categorical values forthe network factor (and any factor under consideration) to a numericalrepresentation (415), for instance, using one-hot-encoding.

A one-hot-encoding is a representation of categorical variables (e.g.,“creative” and “network”) as binary vectors. As an example, thecategorical values can first be mapped to integer values (e.g., fromchar values to integer values). Then, each integer value is representedas a binary vector that is all zero values except the index of theinteger, which is marked with a 1. This integer encoding can then beconverted to a one-hot-encoding of integer encoded values. OneHotEncoderis an example of a one-hot-encoding transformer. One-hot-encoding isknown to those skilled in the art and thus is not further describedherein. Other categorical-to-numerical transformers may also be used.

Once the data structure storing the input data (e.g., Table 7) is readyfor consumption by the weighted GLM, the media creative performanceanalyzer is operable to perform a weighted GLM fitting operation toobtain the impacts of these media creatives beyond the effects of thefactors (e.g., the networks) (305). Here, an objective is to model theexpected value of a continuous variable, Y, as a linear function of thecontinuous predictor, X. To achieve this objective, the weighted GLM isfitted for Y˜X, where X is the set of indicators (factors) for networksand creatives, and the weights w represent the dollars spent pernetwork-media creative combination, holding everything else constant. Inthis way, the weighted GLM can quantify the impact of ad spend on aparticular network and the impact of ad spend on a particular mediacreative.

The concept behind the weighted GLM is to solve for the parameters thatminimize the errors using the Ordinary Least Squares (OLS) methodology.Typically, OLS results in equation 1.1 as follows.y=Xβ is solved as β=(X′X)⁻¹ X′ _(y)  Equation [1.1]

For the weighted GLM, the matrix algebra adjusts this equation andsolves the OLS as follows:wy=wXβ, where w is the weighting vector

Thus, a solution using matrix algebra provides the following:(X′wX)β=X′wy(X′wX)⁻¹(X′wX)β=(X′wX)⁻¹ X′wy

and, as (X′wX)⁻¹(X′wX)=1, this results in equation 1.2 as follows:β=(X′wX)⁻¹ X′wy  Equation [1.2]

Fitting the weighted GLM over the input data prepared by the mediacreative performance analyzer produces coefficients on the mediacreatives under consideration in the form of betas (β). Thesecoefficients reflect the impacts of the media creatives beyond networkeffects.

To debias and correct any difference caused by the “network” factor,each impact can then be utilized to adjust/correct the efficiencycomputed for each media creative (310). In some embodiments, with thecoefficients from the weighted GLM fitting operation, each mediacreative's actual airing efficiency is converted into an adjustedefficiency based on a hypothetical airing of spots to ensure that thesame distribution of network spend is used per media creative. In thisway, the efficiencies of different media creatives will be comparable toeach other.

In some embodiments, the media creative performance analyzer maydetermine how results from the media creative performance analysisshould be visualized or otherwise presented on UIs (315). In someembodiments, the media creative performance analyzer may provide theresults to a visualizer (e.g., visualizer 270) of the quantificationsystem (e.g., quantification system 250) disclosed herein. In turn, thevisualizer may operate to determine how the results should be presentedto users.

For instance, as discussed above, in some cases, media creatives createdfor the same TV campaign may have no or not enough network overlaps(405). However, as illustrated in Table 3, it is possible that pocketsor clusters of network-media creative combinations could be found insome subsets or “islands” of the media creative datasets underconsideration (421).

In some embodiments, finding eligible media creatives for debiasing mayinclude the following steps. First, a definition of nodes and edges iscreated. Nodes represent media creatives. Edges represent the networkson which the media creatives (nodes) are aired. Next, a definition of“islands” is created. In the concept of graphs, “islands” representsubgraphs where all the media creatives are connected directly orindirectly through other media creatives. Media creatives that areeligible for debiasing should belong to the same “island” (subgraph).Media creatives in different islands cannot be compared. Thus, debiasingcan only be performed per island.

If such islands can be found, the datasets can be decoupled intonon-overlapping clusters of groups of networks and media creatives(423). Within each cluster, the input data to the weighted GLM can beprepared (301), the weighted GLM can be fitted to obtain individualmedia creative impacts (305), and the computed media creative impactscan be utilized to adjust the media creative efficiency (310) asdescribed above. As the number of networks increases and the number ofcreatives expands as well, the decoupling can be much more complex, withmany different ‘islands’ of disparate network-media creative clusters.

These different scenarios can be visualized depending upon use case. Forinstance, instead of showing negative relative performance for mediacreatives having no or not enough network overlaps, a “not applicable”or “N/A” message can be displayed. As another example, the results canbe visualized separately for each distinct cluster. An alternativeembodiment might be to present results for the largest cluster (e.g.,most spend).

Other visualization configurations may also be possible. For instance, auser may wish to utilize the media creative efficiency debiasingsolution disclosed herein to make immediate, urgent decisions near realtime (e.g., when a TV network has a fire sale of inventory spots). Sucha user may be a representative of a media agency that produces mediacreatives or any entity that wishes to increase the efficiency andeffectiveness of media creatives. As an example, the user may access aUI generated and provided by a visualizer (e.g., visualizer 270) of thequantification system (e.g., quantification system 250) disclosedherein. The visualizer may operate to compute the percentage of dollarsspent per network over a time period. FIG. 5 is a flow chart thatillustrates this process.

In this example, for every network, i, there is an associated spendpercentage, sp(i). Whatever time period the user has selected throughthe user interface on the user's device (e.g., client device 230 a), thevisualizer is operable to compute and present a distribution of networkspend corresponding to that time period (501).

For instance, if a budget is spent on three TV networks, NA, NB, and NC,during a certain time period, the server computer is operable to computethe percentage of dollar spend for each TV network. Suppose, in thisexample, the distribution of network spend is as follows: sp(NA)=0.5,sp(NB)=0.32, sp(NC)=0.18. The media creative efficiency is computedunder same spending distribution (505). Accordingly, the projectedefficiency equation for a given time period becomes:

${efficiency} = {{1.0 \times {\beta(0)}} + {\sum\limits_{i = 1}^{N_{network}}\;{{\beta(i)} \times {{sp}(i)}}} + {\sum\limits_{j = {1 + N}}^{N + m_{creative}}{{\beta(j)} \times {{sp}(j)}}}}$

Here, the spend percentage, sp(i), is plugged in for each of thenetworks and media creatives. Note that this is a no-interaction casefor networks and media creatives only. This can be extended toadditional factor(s) (510) such as rotations, with no-interaction, asfollows:

${efficiency} = {{1.0 \times {\beta(0)}} + {\sum\limits_{i = 1}^{N_{{network} - {rotation}}}\;{{\beta(i)} \times {{sp}(i)}}} + {\sum\limits_{j = {1 + N}}^{N + m_{creative}}{{\beta(j)} \times {{sp}(j)}}}}$

In some embodiments, the methodology can be extended to rotations alongwith incorporation of interaction effects as follows:

${efficiency} = {{1.0 \times {\beta(0)}} + {\sum\limits_{i = 1}^{N_{{network} - {rotation}}}\;{{\beta(i)} \times {{sp}(i)}}} + {\sum\limits_{j = {1 + N}}^{N + m_{creative}}{{\beta(j)} \times {{sp}(j)}}} + {\sum\limits_{i = 1}^{N_{{network} - {rotation}}}{\sum\limits_{j = {1 + N}}^{N + m_{creative}}{{\beta\left( {i,j} \right)} \times {{sp}\left( {i,j} \right)}}}}}$

Referring back to the no-interaction case for networks and mediacreatives only, the next step is to incorporate the network spendpercentages in the input data and apply the weighted GLM to obtain eachmedia creative efficiency (515). This can be done for creatives 1 . . .k where creative(k) is an indicator for the kth media creative. Thus,for two media creatives 1 and 2, their respective sp(i) can beincorporated in the weighted GLM to obtain the impact of the mediacreative with the computed spend distribution.

${{efficiency}\left( {{creative}\mspace{14mu} 1} \right)} = {{1.0 \times {\beta(0)}} + {\sum\limits_{i = 1}^{N_{network}}{{\beta(i)} \times {{sp}(i)}}} + {{{creative}(1)} \times {\beta\left( {N + 1} \right)}}}$${{efficiency}\left( {{creative}\mspace{14mu} 2} \right)} = {{1.0 \times {\beta(0)}} + {\sum\limits_{i = 1}^{N_{network}}{{\beta(i)} \times {{sp}(i)}}} + {{{creative}(2)} \times {\beta\left( {N + 2} \right)}}}$

The above calculated media creative efficiency can then be adjustedand/or normalize to a media creative efficiency over a time period(520). Here, the total efficiency (efficiency_(tot)) is defined as thetotal lift for the time period divided by the total spend for that timeperiod:efficiency_(tot)=lift_(tot)/spend_(tot)

The overall efficiency can be calculated by summing up the weightedcreative efficiencies. The percentage of network spend can then bedetermined for each media creative.

${overalleffcalc} = {\sum\limits_{i = 1}^{n_{creative}}{{{efficiency}\left( {{creative}\mspace{14mu} k} \right)} \times {{sp}(k)}}}$

This computation can be adjusted by the ratio of the total efficiency tothe computed creative efficiency. This normalizes the value for theadjusted media creative efficiency.efficiency_(adj)(creative k)=efficiency(creativek)×(efficiency_(tot)/overalleffcalc)

In this way, the relative performance of each media creative in thepresence of differences in spend per network (and time aired or daypart)can be isolated and, in some cases, presented to users over a network.Because these computations can be performed on the fly based on a realtime data feed, results from the computations can be provided to usersin near real time to assist in urgent decision making processes, savingtime while providing a visualization of possible outcomes from suchdecisions. Additionally, embodiments disclosed herein improve thegranularity of offline performance analyses from the network level tothe media creative level.

FIG. 6 depicts an example of UI 600 showing individual creativeefficiencies with their relative performance indicators. In thisexample, each creative efficiency is represented by a numerical valuethat is computed by the media creative performance analyzer using themedia creative debiasing methodology described above. This provides anaccurate, unbiased, and quantified measure of effectiveness of eachmedia creative in near real time, which is otherwise not possible for anoffline medium. Historically, media agencies have no visibility onperformance of creatives. Embodiments disclosed herein can bridge thedata gap between the online world and the offline world, bringingoffline processes into the real time decision making with accuratecalculation. Another technical benefit provided by the detailed,creative level of performance analysis is that debiased creativeefficiencies can be leveraged to optimize operations down to individualrotations of creatives based on the cost per sale for each of those.

FIG. 7 depicts a diagrammatic representation of a data processing systemfor implementing a system for processing messages. As shown in FIG. 7,data processing system 700 may include one or more central processingunits (CPU) or processors 701 coupled to one or more user input/output(I/O) devices 702 and memory devices 703. Examples of I/O devices 702may include, but are not limited to, keyboards, displays, monitors,touch screens, printers, electronic pointing devices such as mice,trackballs, styluses, touch pads, or the like. Examples of memorydevices 703 may include, but are not limited to, hard drives (HDs),magnetic disk drives, optical disk drives, magnetic cassettes, tapedrives, flash memory cards, random access memories (RAMs), read-onlymemories (ROMs), smart cards, etc. Data processing system 700 can becoupled to display 706, information device 707 and various peripheraldevices (not shown), such as printers, plotters, speakers, etc. throughI/O devices 702. Data processing system 700 may also be coupled toexternal computers or other devices through network interface 704,wireless transceiver 705, or other means that is coupled to a networksuch as a local area network (LAN), wide area network (WAN), or theInternet.

Those skilled in the relevant art will appreciate that the invention canbe implemented or practiced with other computer system configurations,including without limitation multi-processor systems, network devices,mini-computers, mainframe computers, data processors, and the like. Theinvention can be embodied in a computer or data processor that isspecifically programmed, configured, or constructed to perform thefunctions described in detail herein. The invention can also be employedin distributed computing environments, where tasks or modules areperformed by remote processing devices, which are linked through acommunications network such as a LAN, WAN, and/or the Internet. In adistributed computing environment, program modules or subroutines may belocated in both local and remote memory storage devices. These programmodules or subroutines may, for example, be stored or distributed oncomputer-readable media, including magnetic and optically readable andremovable computer discs, stored as firmware in chips, as well asdistributed electronically over the Internet or over other networks(including wireless networks). Example chips may include ElectricallyErasable Programmable Read-Only Memory (EEPROM) chips. Embodimentsdiscussed herein can be implemented in suitable instructions that mayreside on a non-transitory computer readable medium, hardware circuitryor the like, or any combination and that may be translatable by one ormore server machines. Examples of a non-transitory computer readablemedium are provided below in this disclosure.

ROM, RAM, and HD are computer memories for storing computer-executableinstructions executable by the CPU or capable of being compiled orinterpreted to be executable by the CPU. Suitable computer-executableinstructions may reside on a computer readable medium (e.g., ROM, RAM,and/or HD), hardware circuitry or the like, or any combination thereof.Within this disclosure, the term “computer readable medium” is notlimited to ROM, RAM, and HD and can include any type of data storagemedium that can be read by a processor. Examples of computer-readablestorage media can include, but are not limited to, volatile andnon-volatile computer memories and storage devices such as random accessmemories, read-only memories, hard drives, data cartridges, directaccess storage device arrays, magnetic tapes, floppy diskettes, flashmemory drives, optical data storage devices, compact-disc read-onlymemories, and other appropriate computer memories and data storagedevices. Thus, a computer-readable medium may refer to a data cartridge,a data backup magnetic tape, a floppy diskette, a flash memory drive, anoptical data storage drive, a CD-ROM, ROM, RAM, HD, or the like.

The processes described herein may be implemented in suitablecomputer-executable instructions that may reside on a computer readablemedium (for example, a disk, CD-ROM, a memory, etc.). Alternatively oradditionally, the computer-executable instructions may be stored assoftware code components on a direct access storage device array,magnetic tape, floppy diskette, optical storage device, or otherappropriate computer-readable medium or storage device.

Any suitable programming language can be used to implement the routines,methods, or programs of embodiments of the invention described herein,including Python. Other software/hardware/network architectures may beused. For example, the functions of the disclosed embodiments may beimplemented on one computer or shared/distributed among two or morecomputers in or across a network. Communications between computersimplementing embodiments can be accomplished using any electronic,optical, radio frequency signals, or other suitable methods and tools ofcommunication in compliance with known network protocols.

Different programming techniques can be employed such as procedural orobject oriented. Any particular routine can execute on a single computerprocessing device or multiple computer processing devices, a singlecomputer processor or multiple computer processors. Data may be storedin a single storage medium or distributed through multiple storagemediums, and may reside in a single database or multiple databases (orother data storage techniques). Although the steps, operations, orcomputations may be presented in a specific order, this order may bechanged in different embodiments. In some embodiments, to the extentmultiple steps are shown as sequential in this specification, somecombination of such steps in alternative embodiments may be performed atthe same time. The sequence of operations described herein can beinterrupted, suspended, or otherwise controlled by another process, suchas an operating system, kernel, etc. The routines can operate in anoperating system environment or as stand-alone routines. Functions,routines, methods, steps, and operations described herein can beperformed in hardware, software, firmware, or any combination thereof.

Embodiments described herein can be implemented in the form of controllogic in software or hardware or a combination of both. The controllogic may be stored in an information storage medium, such as acomputer-readable medium, as a plurality of instructions adapted todirect an information processing device to perform a set of stepsdisclosed in the various embodiments. Based on the disclosure andteachings provided herein, a person of ordinary skill in the art willappreciate other ways and/or methods to implement the invention.

It is also within the spirit and scope of the invention to implement insoftware programming or code any of the steps, operations, methods,routines or portions thereof described herein, where such softwareprogramming or code can be stored in a computer-readable medium and canbe operated on by a processor to permit a computer to perform any of thesteps, operations, methods, routines or portions thereof describedherein. The invention may be implemented by using software programmingor code in one or more digital computers, by using application specificintegrated circuits, programmable logic devices, field programmable gatearrays, optical, chemical, biological, quantum or nanoengineeredsystems, components and mechanisms may be used. The functions of theinvention can be achieved in many ways. For example, distributed ornetworked systems, components, and circuits can be used. In anotherexample, communication or transfer (or otherwise moving from one placeto another) of data may be wired, wireless, or by any other means.

A “computer-readable medium” may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, system, ordevice. The computer readable medium can be, by way of example only butnot by limitation, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, system, device,propagation medium, or computer memory. Such computer-readable mediumshall be machine readable and include software programming or code thatcan be human readable (e.g., source code) or machine readable (e.g.,object code). Examples of non-transitory computer-readable media caninclude random access memories, read-only memories, hard drives, datacartridges, magnetic tapes, floppy diskettes, flash memory drives,optical data storage devices, compact-disc read-only memories, and otherappropriate computer memories and data storage devices. In anillustrative embodiment, some or all of the software components mayreside on a single server computer or on any combination of separateserver computers. As one skilled in the art can appreciate, a computerprogram product implementing an embodiment disclosed herein may compriseone or more non-transitory computer readable media storing computerinstructions translatable by one or more processors in a computingenvironment.

A “processor” includes any, hardware system, mechanism or component thatprocesses data, signals or other information. A processor can include asystem with a central processing unit, multiple processing units,dedicated circuitry for achieving functionality, or other systems.Processing need not be limited to a geographic location, or havetemporal limitations. For example, a processor can perform its functionsin “real-time,” “offline,” in a “batch mode,” etc. Portions ofprocessing can be performed at different times and at differentlocations, by different (or the same) processing systems.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having,” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,product, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, product, article, orapparatus.

Furthermore, the term “or” as used herein is generally intended to mean“and/or” unless otherwise indicated. For example, a condition A or B issatisfied by any one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present). As used herein, a termpreceded by “a” or “an” (and “the” when antecedent basis is “a” or “an”)includes both singular and plural of such term (i.e., that the reference“a” or “an” clearly indicates only the singular or only the plural).Also, as used in the description herein, the meaning of “in” includes“in” and “on” unless the context clearly dictates otherwise. The scopeof the present disclosure should be determined by the following claimsand their legal equivalents.

What is claimed is:
 1. A method for debiasing media creative efficiency,comprising: retrieving, by a computer, offline data from a database, theoffline data including spot airing data comprising information on mediacreatives and networks on which the media creatives aired through lineartelevision, the information containing biases due to a plurality offactors including differences in the networks and audiences of thenetworks; creating, by the computer, a graph representing the mediacreatives connected via the networks; determining, by the computer basedat least on the media creatives and the networks in the spot airingdata, a subgraph of the graph, the subgraph representing a plurality ofmedia creatives with overlapping networks on which the plurality ofmedia creatives aired so as to allow for a media creative performanceanalysis; conducting, by the computer, the media creative performanceanalysis on the plurality of media creatives, the media creativeperformance analysis comprising: computing a media creative efficiencyfor each media creative of the plurality of media creatives per eachnetwork of the overlapping networks; creating a data structure for inputto a weighted generalized linear model (GLM), wherein each entry in thedata structure represents a network-media creative combination derivedfrom the plurality of media creatives and the overlapping networks inthe subgraph; modifying the data structure to include a network spendfor each network-media creative combination; computing a media creativeefficiency for each respective network-media creative-network spendcombination in the data structure, wherein the media creative efficiencyis defined in terms of an incremental lift in unique visitors to awebsite per the network spend on the respective network-media creativecombination; and performing a weighted GLM fitting operation over inputdata in the data structure to obtain quantified impacts of the pluralityof media creatives beyond effects of the overlapping networks and thedifferences in the networks and the audiences of the networks;adjusting, by the computer, the media creative efficiency for eachnetwork-media creative-network spend combination in the data structureutilizing the quantified impacts of the plurality of media creatives tothereby debias the media creative efficiency for each network-mediacreative-network spend combination in the data structure; andpresenting, by the computer through a user interface, results from theadjusting, the presenting including generating a visualization fordisplaying the results on the user interface, the results including theplurality of media creatives and corresponding debiased media creativeefficiencies as adjusted and stored in the data structure.
 2. The methodaccording to claim 1, wherein determining the plurality of mediacreatives further comprises determining eligible media creatives fordebiasing, comprising: creating a definition of nodes representing themedia creatives in the spot airing data and edges representing thenetworks on which the media creatives aired; creating a definition ofislands, each island representing a subgraph of the nodes connecteddirectly or indirectly through the edges; determining whether any islandis present in the media creatives; and responsive to an island beingpresent in the media creatives, returning a set of nodes associated withthe island as the plurality of media creatives.
 3. The method accordingto claim 1, further comprising: prior to performing the weighted GLMfitting operation, transforming categorical values associated with theoverlapping networks into numerical values.
 4. The method according toclaim 1, further comprising: automatically decoupling datasets in thespot airing data into non-overlapping clusters of groups of networks andmedia creatives; and for each cluster, performing the weighted GLMfitting operation to obtain individual impacts of media creatives andadjusting a media creative efficiency previously computed for each mediacreative in the cluster to thereby debias the media creative efficiencyfor the each media creative in the cluster.
 5. The method according toclaim 4, further comprising: generating a visualization for presentationon a user device, the visualization showing relative performance ofmedia creatives across the non-overlapping clusters.
 6. The methodaccording to claim 4, further comprising: generating a visualization forpresentation on a user device, the visualization showing relativeperformance of media creatives in one of the non-overlapping clusters.7. The method according to claim 1, further comprising: generating avisualization for presentation on a user device, the visualizationshowing relative performance of the set of media creatives.
 8. A systemfor debiasing media creative efficiency, comprising: a processor; anon-transitory computer-readable medium; and stored instructionstranslatable by the processor for: retrieving offline data from adatabase, the offline data including spot airing data comprisinginformation on media creatives and networks on which the media creativesaired through linear television, the information containing biases dueto a plurality of factors including differences in the networks andaudiences of the networks; creating a graph representing the mediacreatives connected via the networks; determining, based at least on themedia creatives and the networks in the spot airing data, a subgraph ofthe graph, the subgraph representing a plurality of media creatives withoverlapping networks on which the plurality of media creatives aired soas to allow for a media creative performance analysis; conducting themedia creative performance analysis on the plurality of media creatives,the media creative performance analysis comprising: computing a mediacreative efficiency for each media creative of the-plurality of mediacreatives per each network of the overlapping networks, wherein themedia creative efficiency is defined in terms of an incremental lift inunique visitors to a website per dollar spent on a respective mediacreative; creating a data structure for input to a weighted generalizedlinear model (GLM), wherein each entry in the data structure representsa network-media creative combination derived from the plurality of mediacreatives and the overlapping networks in the subgraph; modifying thedata structure to include a network spend for the each network-mediacreative combination; computing a media creative efficiency for eachrespective network-media creative-network spend combination in the datastructure, wherein the media creative efficiency is defined in terms ofan incremental lift in unique visitors to a website per the networkspend on the respective network-media creative combination; andperforming a weighted GLM fitting operation over input data in the datastructure to obtain quantified impacts of the plurality of mediacreatives beyond effects of the overlapping networks and the differencesin the networks and the audiences of the networks; adjusting the mediacreative efficiency for each network-media creative-network spendcombination in the data structure utilizing the quantified impacts ofthe plurality of media creatives to thereby debias the media creativeefficiency for each network-media creative-network spend combination inthe data structure; and presenting, through a user interface, resultsfrom the adjusting, the presenting including generating a visualizationfor displaying the results on the user interface, the results includingthe plurality of media creatives and corresponding debiased mediacreative efficiencies as adjusted and stored in the data structure. 9.The system of claim 8, wherein determining the plurality of mediacreatives further comprises determining eligible media creatives fordebiasing, comprising: creating a definition of nodes representing themedia creatives in the spot airing data and edges representing thenetworks on which the media creatives aired; creating a definition ofislands, each island representing a subgraph of the nodes connecteddirectly or indirectly through the edges; determining whether any islandis present in the media creatives; and responsive to an island beingpresent in the media creatives, returning a set of nodes associated withthe island as the plurality of media creatives.
 10. The system of claim8, wherein the stored instructions are further translatable by theprocessor for: prior to performing the weighted GLM fitting operation,transforming categorical values associated with the overlapping networksinto numerical values.
 11. The system of claim 8, wherein the storedinstructions are further translatable by the processor for:automatically decoupling datasets in the spot airing data intonon-overlapping clusters of groups of networks and media creatives; andfor each cluster, performing the weighted GLM fitting operation toobtain individual impacts of media creatives and adjusting a mediacreative efficiency previously computed for each media creative in thecluster to thereby debias the media creative efficiency for the eachmedia creative in the cluster.
 12. The system of claim 11, wherein thestored instructions are further translatable by the processor for:generating a visualization for presentation on a user device, thevisualization showing relative performance of media creatives across thenon-overlapping clusters.
 13. The system of claim 11, wherein the storedinstructions are further translatable by the processor for: generating avisualization for presentation on a user device, the visualizationshowing relative performance of media creatives in one of thenon-overlapping clusters.
 14. The system of claim 8, wherein the storedinstructions are further translatable by the processor for: generating avisualization for presentation on a user device, the visualizationshowing relative performance of the set of media creatives.
 15. Acomputer program product for debiasing media creative efficiency, thecomputer program product comprising a non-transitory computer-readablemedium storing instructions translatable by a processor for: retrievingoffline data from a database, the offline data including spot airingdata comprising information on media creatives and networks on which themedia creatives aired through linear television, the informationcontaining biases due to a plurality of factors including differences inthe networks and audiences of the networks; creating a graphrepresenting the media creatives connected via the networks;determining, based at least on the media creatives and the networks inthe spot airing data, a subgraph of the graph, the subgraph representinga plurality of media creatives with overlapping networks on which theplurality of media creatives aired so as to allow for a media creativeperformance analysis; conducting the media creative performance analysison the plurality of media creatives, the media creative performanceanalysis comprising: computing a media creative efficiency for eachmedia creative of the plurality of media creatives per each network ofthe overlapping networks; creating a data structure for input to aweighted generalized linear model (GLM), wherein each entry in the datastructure represents a network-media creative combination derived fromthe plurality of media creatives and the overlapping networks in thesubgraph; modifying the data structure to include a network spend forthe each network-media creative combination; computing a media creativeefficiency for each network-media creative-network spend combination inthe data structure, wherein the media creative efficiency is defined interms of an incremental lift in unique visitors to a website per thenetwork spend on the respective network-media creative combination; andperforming a weighted GLM fitting operation over input data in the datastructure to obtain quantified impacts of the plurality of mediacreatives beyond effects of the overlapping networks and the differencesin the networks and the audiences of the networks; adjusting the mediacreative efficiency for each network-media creative-network spendcombination in the data structure utilizing the quantified impacts ofthe plurality of media creatives to thereby debias the media creativeefficiency for each network-media creative-network spend combination inthe data structure; and presenting, through a user interface, resultsfrom the adjusting, the presenting including generating a visualizationfor displaying the results on the user interface, the results includingthe plurality of media creatives and corresponding debiased mediacreative efficiencies as adjusted and stored in the data structure. 16.The computer program product of claim 15, wherein determining theplurality of media creatives further comprises determining eligiblemedia creatives for debiasing, comprising: creating a definition ofnodes representing the media creatives in the spot airing data and edgesrepresenting the networks on which the media creatives aired; creating adefinition of islands, each island representing a subgraph of the nodesconnected directly or indirectly through the edges; determining whetherany island is present in the media creatives; and responsive to anisland being present in the media creatives, returning a set of nodesassociated with the island as the plurality of media creatives.
 17. Thecomputer program product of claim 15, wherein the instructions arefurther translatable by the processor for: automatically decouplingdatasets in the spot airing data into non-overlapping clusters of groupsof networks and media creatives; and for each cluster, performing theweighted GLM fitting operation to obtain individual impacts of mediacreatives and adjusting a media creative efficiency previously computedfor each media creative in the cluster to thereby debias the mediacreative efficiency for the each media creative in the cluster.
 18. Thecomputer program product of claim 17, wherein the instructions arefurther translatable by the processor for: generating a visualizationfor presentation on a user device, the visualization showing relativeperformance of media creatives across the non-overlapping clusters. 19.The computer program product of claim 17, wherein the instructions arefurther translatable by the processor for: generating a visualizationfor presentation on a user device, the visualization showing relativeperformance of media creatives in one of the non-overlapping clusters.20. The computer program product of claim 15, wherein the instructionsare further translatable by the processor for: generating avisualization for presentation on a user device, the visualizationshowing relative performance of the set of media creatives.